Sensor and imaging system

ABSTRACT

A system is disclosed having a camera, a processor, and a user interface. The camera transmits image data responsive to a scene within a field of view. In response to the image data, the processor indicates whether a condition has been satisfied. The user interface is operably connected to the processor and allows a user to select criteria for detection of objects, for indicating criteria selected, and for providing visual confirmation that an object has been detected.

RELATED APPLICATIONS

This Application is a U.S. National filing under §371 of InternationalApplication No. PCT/US/27351, filed Aug. 31, 2001, claiming priorityfrom U.S. Ser. No. 60/229,613, filed Aug. 31, 2000 (which is herebyincorporated by reference).

TECHNICAL FIELD

The present invention relates to sensor and imaging systems, and moreparticularly to a system for providing and interpreting image data.

BACKGROUND OF THE INVENTION

Sensor and imaging systems are increasingly in demand in today'stechnology driven economy. These systems include a camera for viewingobjects included within a field of view. The camera generates image datathat is analyzed by a computer to determine what, if any, action shouldbe taken in response to the object detected.

Many recognition systems use two or more cameras for viewing objectsincluded within one field of view. In addition to the costs associatedwith using several cameras, these systems require a specific mountingarrangement for each of the cameras. Such systems have a reduced levelof reliability over single camera systems because both cameras areneeded for proper operation.

Single camera systems are typically mounted at a fixed location and lookfor objects that satisfy, or fail to satisfy, predetermined criteria.For instance—systems that check for structural defects. These systemsare incapable of making decisions that are not already specified.

Accordingly, a need exists for a sensor and imaging system that, byusing an image provided by a camera, can decide whether a condition has,or has not, been satisfied.

SUMMARY OF THE INVENTION

In an embodiment in accordance with the present invention, a system isprovided having a camera, a processor, and a user interface. The cameratransmits image data responsive to a scene within a field of view. Inresponse to the image data, the processor indicates whether a conditionhas been satisfied. The user interface is operably connected to theprocessor and allows a user to select criteria for detection of objects,for indicating criteria selected, and for providing visual confirmationthat an object has been detected.

In another embodiment, a control interface is also provided foreffecting other devices. Further, the system provides signals toinfluence other devices.

In yet another embodiment, the system provides a signal to open a doorupon a determination by the processor that a condition has beensatisfied. The door is then open by a conventional electro mechanicaldoor opener system having a drive motor operably connected to the door.

Other features and advantages of the invention will be apparent from thefollowing specification taken in conjunction with the followingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 a and 1 b are a block diagram of an embodiment of a sensor andimaging system in accordance with the present invention;

FIG. 2 is a simplified schematic of another embodiment of a sensor andimaging system in accordance with the present invention and having apair of camera assemblies connected to a pair of sensor/imagingcircuits, respectively;

FIG. 3 is an elevation view of a closed door having the cameraassemblies of FIG. 2 mounted in proximity thereto;

FIG. 4 is similar to FIG. 3 except the door is open;

FIG. 5 is a perspective partial assembly view of one of the cameraassemblies of FIG. 2 for attachment to a header above the door of FIG.3;

FIG. 6 is a cross-sectional view of the camera assembly of FIG. 5;

FIG. 7 is a plan view of a heater assembly attached to the window of thecamera assembly housing of FIG. 5;

FIG. 8 is a cross-sectional view of the heater assembly, and window,taken along plane A-A of FIG. 7;

FIG. 9 is a simplified schematic of the camera within the cameraassembly of FIG. 8 connected to one of the video processors of FIG. 2,

FIG. 10 is a top, perspective view, of one side of the door of FIGS. 3and 4 wherein one of the camera assemblies of FIG. 2 has a field of viewthat includes the area in front of the door;

FIG. 11 is a graphical illustration of the pixel density of the field ofview within FIG. 10 as objects are placed further away from the cameraassembly;

FIG. 12 provides a plurality of safety zone configurations that can, inan embodiment, be selected by a user using the membrane keypad of FIG.2;

FIG. 13 is a plan view of an embodiment of the membrane keypad depictedin FIG. 2;

FIG. 14 is a plan view of a zone positioning method using the membranekeypad of FIG. 13;

FIGS. 15 a and 15 b are a simplified block diagram of one of the objectsensor/imaging circuits of FIG. 2 having a Field Programmable Gate Array(FPGA) and a Central Processing Unit (CPU);

FIG. 16 is a top-level simplified block diagram of a processing systemin accordance with the present invention, having an image processingmodule with the FPGA of FIGS. 15 aand 15 b, a feature processing moduleexecuted by the CPU of FIGS. 15 a and 15 b, and a detection processingmodule executed by the CPU of FIGS. 15 a and 15 b;

FIG. 17 is a simplified block diagram of an initialization sequence forthe processing system of FIG. 16, for initializing input parameters andcalculating related, derived parameters, for initializing detectionthreshold tables, and for initializing detection and feature calculationzones;

FIG. 18 is a simplified block diagram of the initialization sequence forthe processing system of FIG. 16, for initializing the FPGA imageprocessing thresholds and video digitizer settings from user input datavia the FPGA serial input/output board, and for initializing the featureand detection processing systems;

FIG. 19 is a simplified block diagram of a video loop processingsequence for the processing system of FIG. 16, utilizing a ping-pongbuffer to point to, load and unpack reference images into user data;

FIG. 20 is a simplified block diagram of the video loop processingsequence for the processing system of FIG. 16, utilizing user data inthe CPU to generate features and detection decisions on a current imageframe;

FIG. 21 is a simplified block diagram of the video loop processingsequence for the processing system of FIG. 16, having diagnosticmessages outputted at the end of each frame processing and at any pointin the video processing;

FIG. 22 is a simplified block diagram of a ping-pong system inaccordance with the present invention;

FIG. 23 is a simplified block diagram of the initialization of theping-pong system depicted in FIG. 22;

FIG. 24 is a simplified block diagram of the ping-pong process loop forthe ping-pong system of FIG. 22;

FIG. 25 is a simplified block diagram of an automatic contrastcompensation initialization for the processing system of FIG. 16, forchanging the video gain in response to image characteristic criteria andtime dynamic criteria;

FIG. 26 is a simplified block diagram of the automatic contrastcompensation in a video loop for the processing system of FIG. 16, forchanging the video gain in response to image characteristic criteria andtime dynamic criteria;

FIG. 27 is a simplified block diagram of the zone initializationsequence for the system initialization of FIG. 17, for building zones inreal world coordinates, generating zone submasks and constructing zonemasks;

FIG. 28 is a simplified block diagram of the threshold tablesinitialization sequence for the system initialization of FIG. 17, havingan initialize camera intrinsic parameters module, a resolution modelmodule and an object model module;

FIG. 29 is a simplified block diagram of the image processing module forthe processing system of FIG. 16, having an edge detector;

FIG. 30 is a simplified block diagram of the image processing module forthe processing system of FIG. 16, having a motion detector that usesregions;

FIG. 31 is a simplified block diagram of the image processing module forthe processing system of FIG. 16, having region analysis for shadow andlightbeam processing;

FIG. 32 is a simplified block diagram of the image processing module forthe processing system of FIG. 16, having a motion detector that usesedges;

FIG. 33 is a simplified block diagram of the feature processing modulefor the processing system of FIG. 16, for calculating presence, motion,frame and region features;

FIG. 34 is a simplified block diagram of the feature generation systemof FIG. 33, having a label module, a calculate global presence featuresmodule and a calculate zone presence features module;

FIG. 35 is a simplified block diagram of the calculate presence featuresystem of FIG. 34, for calculating global presence features;

FIG. 36 is a simplified block diagram of the calculate presence featuresystem of FIG. 34, for calculating zone presence features;

FIG. 37 is a simplified block diagram of the feature generation systemof FIG. 33, having a label module, a calculate global shadow andlightbeam features module, and a calculate shadow and lightbeam zonefeatures module;

FIG. 38 is a simplified block diagram of the calculate shadow andlightbeam region features system of FIG. 37, for calculating globalshadow and lightbeam features;

FIG. 39 is a simplified block diagram of the calculate shadow andlightbeam region features system of FIG. 37, for calculating shadow andlightbeam zone features;

FIG. 40 is a simplified block diagram of a split histogram grey levelanalysis for the processing system of FIG. 16;

FIG. 41 is a simplified block diagram of the feature generation systemof FIG. 33, for calculating frame features;

FIG. 42 is a simplified block diagram of the information processingmodule of FIG. 16, for detecting the presence of an object from featuresgenerated;

FIG. 43 is a simplified block diagram of the information processingmodule of FIG. 16, for evaluating and updating reference images;

FIG. 44 is a simplified block diagram of the information processingmodule of FIG. 16, for changing threshold values relative to changingbackground values from the field of view;

FIG. 45 is a simplified block diagram of the information processingmodule of FIG. 16, for determining the geometric association of edge andzone data in a detected object;

FIG. 46 is a continuation of FIG. 45 for the simplified block diagram ofthe information processing module of FIG. 16, and includes testingregion edges to determine zone intersections in detected objects;

FIG. 47 is a continuation of FIG. 46 for the simplified block diagram ofthe information processing module of FIG. 16, and includes evaluatingregion scores of zones to determine zone intersections in detectedobjects;

FIGS. 48 and 49 are tables for defining various image operations;

FIG. 50 is a representative example of a background or reference image;

FIG. 51 is similar to FIG. 50 except it is a current image wherein anobject has entered the field of view;

FIG. 52 is the difference between subtracting the reference image inFIG. 50 from the current image in FIG. 51;

FIG. 53 is the difference between subtracting the current image in FIG.51 from the reference image in FIG. 50;

FIG. 54 is a resulting image after thresholding and shape filtering theimage in FIG. 52;

FIG. 55 is a resulting image after thresholding and shape filtering theimage in FIG. 53;

FIG. 56 is a resulting image after completing a logical OR operation onthe images in FIG. 54 and FIG. 55; and

FIG. 57 is similar to FIG. 56 except that regions within the image ofFIG. 56 are labeled for classification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

While this invention is susceptible of embodiments in many differentforms, there is shown in the drawings and will herein be described indetail preferred embodiments of the invention with the understandingthat the present disclosure is to be considered as an exemplification ofthe principles of the invention and is not intended to limit the broadaspect of the invention to the embodiments illustrated.

Turning to FIGS. 1 a and 1 b, a block diagram of an embodiment of asystem in accordance with the present invention is depicted. The system10 includes aspects directed to image analysis 12, presence detection14, sensor technology 16, electronics 18, device control 20, camera pose22, camera technology 24, image analysis 26, and market application 28.

Turning to FIG. 2, a simplified schematic of another embodiment of asystem in accordance with the present invention is depicted. The system110 includes a pair of camera assemblies 112,113 operably connected,respectively, to a pair of sensor/imaging circuits 114,115. Thesensor/imaging circuits 114,115 are operably connected to an I/O displayboard 116 that is connected to a keypad 118. In a further embodiment,the system 110 can include an electro mechanical door opener system 117having a drive motor 119 operably connected to a door 121 (FIGS. 3 and4).

The cameras assemblies 112 and 113 can include charge coupled devices(CCD), or the like, having preferably a wide-angle lens, and capable oftransmitting image data to the sensor/imaging circuits 114 and 115,respectively. The image data corresponds, respectively, to a scenewithin each camera's field of view.

The sensor/imaging circuits 114,115 process the image data fordetermining whether a user selected condition has been satisfied. Theuser selected conditions are selected via a man-machine interfacecomprising the I/O display board 116 and the membrane keypad 118. In anembodiment, the man-machine interface is operably connected to thesensor/imaging circuits 114,115 and allow a user to select criteria fordetection of objects, for indicating criteria selected, and forproviding visual confirmation that an object has been detected.

FIG. 3 provides a cross sectional elevation view of a doorway 120 thatis closed by a door 121. Mounted in proximity to the doorway 120 and onopposite sides thereof are the camera assemblies 112,113 of FIG. 2.According, the camera assemblies 112 and 113 have a field-of-view 122and 123, respectively, on opposite sides of the doorway. Preferably, thefield-of-view of at least one camera assembly includes the doorway 120when the door 121 is open as shown in FIG. 4. Further, thefield-of-views 122 and 123 overlap about the doorway 120.

Turning to FIG. 5 a perspective partial assembly view is depicted of oneof the camera assemblies 112,113 of FIG. 2 for attachment to a headerabove the door 121 of FIG. 3. The camera assembly includes a housingassembly 131 having a window 132. FIG. 6 provides a cross-sectional viewof the camera assembly of FIG. 5. Mounted within the housing assembly131 is a camera 134 having a field-of-view 136 and a heater assembly140. In a further embodiment, a power supply (not shown) can be mountedwithin the heater assembly for supplying power to the camera 134.

FIGS. 7 and 8 provide a plan view and a cross-sectional view,respectively, of the heater assembly 140. In an embodiment, the heaterassembly 140 adjoins the window 132 of the camera assembly housing 131,preferably made of cast metal or the like, and includes a ceramicresistor 142 sandwiched between a pair of conductive terminals 144,145.The ceramic resistor 142 is generally ring shaped and constructed of amaterial that exhibits what is commonly referred to as a PTCR(positive-temperature coefficient or resistance) or PTC effect.Likewise, the conductive terminals 144,145 are generally ring shaped andconstructed of an electrically and thermally conductive material such ascopper.

In an embodiment, annular apertures 146 extend through the axis of theceramic resistor 142 and the conductive terminals 144,145. The apertures146 have substantially identical outer circumferences and areconcentrically aligned with each other.

Preferably, the outer perimeter 148 of conductive terminal 145 includesa plurality of ears 150 extending outwardly therefrom. Extending througheach ear 150 is an aperture 152 for extending an attachment screw 154(FIG. 9) therethrough.

Turning to FIG. 9, the heater assembly 140 is mounted within the housing131 of the camera assembly. As indicated previously, attachment screws154 couple the conductive terminal 145 of the heater assembly 140 to amounting surface, or alternatively, mounting posts 156 that inwardlyextend into the housing 131 and are integrally attached thereto.

The camera assembly housing 131 includes an aperture 158 that allows thecamera's field-of-view 136 to extend outside of the housing. The window132 is mounted over the aperture 158 to prevent contaminants such asdirt and moisture from entering the camera assembly.

Preferably, the window 132 is sandwiched between the thermallyconductive terminal ring 145 of the heater assembly 140 and an annulargasket 160 made of a resilient material and adjoining against the innersurface 162 of the camera assembly housing 131 about aperture 158. In anembodiment, the window 132 is made of a visually transparent materialsuch as borosilicate glass.

The camera 134 generates image data or electronic pixel data 218representative of what is observed in the camera's field-of-view 136. Inan embodiment, the image data 218 is analyzed by one of the videoprocessors 114,115 (FIG. 2) for determining what, if any, action shouldbe taken in response to what is detected in the camera's field-of-view136.

Within the camera assembly housing 131, the terminals 144,145 of theheater assembly 140 are coupled to a voltage source 166 for maintaininga voltage potential across the ceramic resistor 142. The heat generatedby the ceramic resistor 142 as current flows therethrough is dissipatedthrough conductive terminal 145 and onto the window 132. In anembodiment, enough heat is provided to maintain the window 132 at atemperature above the dew-point of the air outside of the cameraassembly housing 131. For instance, the heater can activate at about 87°F. and deactivate at about 106° F.

As will be appreciated by those having skill in the art, the use of aceramic resistor 142 eliminates the need for a mechanical thermostat orthe like since the resistor material exhibits a positive-temperaturecoefficient. The elimination of a thermostat increases the reliabilityof the heater and reduces the amount of noise placed on voltage suppliesas a result of switching. Turning back to FIG. 2, in an embodiment thecamera assemblies 112,113 can include an illumination source (notshown), such as a lightbulb, light emitting diodes within the humanlyvisible or non-visible spectrum, or the like, to illuminate the field ofview. The output of the camera assemblies 112,113 can be received by aremovably coupled display (not shown) for allowing a user to view checkthat each camera assemblies' field of view is properly configured.

Turning to FIG. 10 a top perspective view is provide of one side of thedoor 121 of FIGS. 3 and 4 wherein one of the camera assemblies (inparticular camera assembly 112) of FIG. 2 has a field of view 122 thatincludes the area in front of the door. The camera assembly 122 providesimage data to the video processor 114 (FIG. 2) which, in an embodiment,superimposes a safety zone 168 and, if desired, one or more activationzones 170. Preferably, when an object is detected by either of the videoprocessors 114,115 to be in a safety zone 168, the door 121 is openedand remains so until the object is outside of the safety zone 168.Further, when an object is detected by either of the video processor114,115 to enter an activation zone 168, the door 121 is opened areremains open for a period of time after the object stops moving in theactivation zone or leaves the activation zone.

Preferably, the safety zone 168 is maintained in an area immediatelysurrounding the door 121 to prevent the door from closing when a personor object is in the immediate vicinity of the door. Moreover, theactivation zone(s) 168 open the door when a person or vehicle approachesthe door 121. A failsafe system can also be provided to open the door121 whenever there is a loss of illumination within the field of view,severe illumination changes, electronics failure, camera knocked ajar,or the camera lens is obscured.

FIG. 11 depicts the pixel density of the field of view 122 as objectsare placed further away from the camera assembly 122. FIG. 12 provides aplurality of safety zone configurations that can, in an embodiment, beselected by a user using the membrane keypad 118 (FIG. 2).

FIG. 13 provides an plan view of an embodiment of the membrane keypad118 depicted in FIG. 2. The membrane keypad 118 and the I/O displayboard 116 (FIG. 2) provide a user interface or man-machine interface fora user to setup the system (FIG. 2). In an embodiment, the userinterface allows a user to define the door type, and dimensions, andestablish the size, location, and type of detection zones the system 110will use. Preferably, the user interfaces allows a user to enter data todefine at least one control zone parameter (i.e., activation zone orsafety zone) from the group consisting of area, location, shape, numberof control zones, and control criteria.

As indicated previously, the system 110, via the user interface, has thecapability to define at least a portion of an image received by thecamera assemblies as a control zone. In an embodiment, the system 110has the capability to choose coordinates from all pixel coordinated bydirect access within the control zone. Alternatively, the system 110 hasthe capability to choose from multiple predefined zones. Moreover, thesystem 110 can have the capability to put real objects in the field ofview so as to delineate boundary coordinates and the real objects becomepart of the image data.

In an embodiment, the user interface has three modes of operation:parameter edit mode, run mode, and diagnostic mode. In parameter editmode, a user can input or modify configuration parameters, using touchkeypad buttons 610, 622, 624, and 626, such as the door model, Englishor metric units, camera heights and distance from the door. In the runmode, the system 110 is activated. As such, the system 110 processesimages from the cameras 112,113 and outputs safety and activation zoneindication signals through the I/O board 116, and displays statusinformation on the display LEDs 614, and 616. In the diagnostic mode,additional information regarding the status of the system 110 is madeavailable via an I/O port (not shown).

FIG. 14 provides a diagram depicting movement of zones using themembrane keypad of FIG. 13. In an embodiment, a user can move activationzones to various locations within the field-of-view of the cameraassemblies 112,113.

FIG. 15 provides a simplified block diagram of one of the sensor/imagingcircuits 114,115 of FIG. 2. In an embodiment, the sensor/imaginingcircuits 114,115 are substantially similar in physical design andinclude a Field Programmable Gate Array (FPGA) 124, a Central ProcessingUnit (CPU) 125, and a video digitizer 126.

In an embodiment, the video digitizer 126 receives an analog imagesignal from one of the cameras, digitizes the analog image signal, andtransmits the digitized image signal to the field programmable gatearray 124.

As explained in detail further herein, the field programmable gate array124 is programmed to perform one or more image processing operations inresponse to the digitized image signal received. In an embodiment, theseoperations include comparing predetermined traits of the digitized imagesignal with one or more previously received digitized image signals toprovide composite image data. In response to the composite image data,the central processing unit 125 determines whether one or moreconditions have been satisfied.

Operably coupled to the video digitizer 126 is a temperature sensor 128having an output correlating to the temperature of the video digitizer.Upon an indication by the temperature sensor of a condition whereby thevideo digitizer 126 is not operating within a predetermined temperaturerange or limit, reset command is issued by a reset circuit 130 wherebythe system 110 maintained in a reset state until the temperature of thevideo digitizer returns to within the predetermined temperature range orlimit.

In an embodiment, the FPGA 124 performs a relatively high-rate pixelprocessing in order to unburden the CPU 125 and achieve a desired videoprocessing frame rate. This hardware architecture balance reducesoverall system cost by removing the cost associated with an adequatelyfast CPU chip. A further frame rate speed up can be achieved by usingthe FPGA and CPU processing simultaneously in parallel. This parallelprocessing is accomplished by FPGA pixel processing the next frameduring the interval that the CPU is data processing the current frame.Thus, the new FPGA output is immediately available to the CPU processwhen the CPU finishes the current frame data processing. This processstructure requires the ability to maintain two independent sets of data,and is referred to later herein as ping/pong control.

Turning to FIG. 16, a top-level simplified block diagram is depicted ofa processing system in accordance with the present system. The systempreferably includes an image processing module 212 within the FPGA 124of FIGS. 15 a and 15 b, a feature processing module 214 executed by theCPU 125 of FIGS. 15 a and 15 b , and a detection processing module 216also executed by the CPU 125 of FIGS. 15 a and 15 a.

In an embodiment, the image processing module 212 receives storedelectronic pixel data 218 comprising current image data 220, referenceimage data 222, reference edges data 224, and previous image data 226.Preferably, the current image data 220 is the most recently taken image(i.e., taken at t), the previous image data 226 is the next mostrecently taken image data (i.e., taken at t+1), and the reference imagedata 222 is the oldest of the taken image data (i.e., taken at t+1+x).Moreover, as explained in detail further herein, the reference edgesdata 224 consists of edge data extracted from the reference image data222.

The image processing module 212 also receives parameter data 228 fromthe man-machine interface (i.e., membrane keypad 118 and I/O displayboard 116 of FIG. 2). As explained in detail further herein, theparameter data 228 includes information pertaining what areas (i.e.,control zones) that a detected object within the field of views (112 and123 of FIGS. 2 and 3) is to result in opening of the door 121 (FIGS. 2and 3).

As explained in detail further herein, in response to the electronicpixel data 218 and the parameter data 228, the image processing module212 produces derived image data 230 comprising edge segmentation, motionsegmentation, and region segmentation.

The feature processing module 214 receives the derived image data 230and the parameter data 228. As explained in detail further herein, thefeature processing module 214 produces, in response to the image data230 and parameter data 228, feature data 232 comprising edge features,motion features, region features, and frame features.

The detection processing module 216 receives the feature data 232 andthe parameter data 228. In response to the data, the detectionprocessing module 216 produces control signals 234 comprising adetection signal for opening and closing the door 121 (FIGS. 2 and 3),reference updates, and adaptive thresholds.

Turning to FIGS. 17 and 18, a simplified block diagram is depicted of aninitialization sequence for the processing system of FIG. 16. Theinitialization sequence 234 includes an initialize parameters step 236,an initialize threshold tables step 238, an initialize zones step 240,an initialize FPGA step 242, an initialize video digitizer step 244, andan initialize video system step 246.

The initialize parameters step 236 includes initialization of theman-machine interface and constant data and derived parameters. Duringinitialization of the man-machine interface, user entered data is readand stored into memory. Constant data is also loaded into memory alongwith derived parameters relating to control zones for opening andclosing the door.

Thus, as indicated above, upon application of power to the system, theinitialize parameter module 236 initiates the initialization of theman-machine interface (i.e., membrane keypad 118 and I/O display board116 of FIG. 2), constant data, and derived parameters. The initializethreshold table module 238 initiates the initialization of the areathreshold maps from the camera geometry and resolution models. Thesemaps are used to determine minimum and maximum pixel characteristics ofobjects such as people and vehicles such as, for example, forklifts.

The initialize zones module 234 initiates the initialization of thecontrol zones whereupon data associated with user or predefined safetyzones and activation zones is complied. The initialize FPGA 242 and theinitialize video digitizer 244 initiates the initialization of the FPGA124 FIGS. 15 a and15 b) and the video digitizer 126 (FIGS. 15 a and 15b), respectively. In particular, the control status register (CSR) andimage buffers pointer are initialized during FPGA initialization.Further, the video digitizer is initialized by constructing the requiredinstructions and sending them, via the FPGA serial I/O.

The initialize video system 246 initiates the initialization of the CPU125 (FIGS. 15 a and 15 b). In particular, the first ping-pong data setis selected. Next, the FPGA is instructed to capture a video frame. Fourimages (reference images) are then initialized—a grey level reference(R), an edge reference (GER) and an archive edges reference. The FPGA isthen instructed to process these data. The FPGA outputs are retrievedfrom the FPGA into the Bank 0 database.

After the system is initialized, the system operates in a videoprocessing loop depicted in the simplified block diagrams of FIGS.19-22. In an embodiment, the video processing loop 250 includes a getnext frame step 252, a retrieve FPGA data step 254, a calculate allfeatures step 214, a calculate detection & control step 258, a save FPGAdata step 260, and a write diagnostics step 262.

Within the video processing loop 250, the CPU 125 process use thecurrent ping/pong buffer to point to, load and unpack that data into athird database—the user data—bank U. This data is used in the CPUprocess later to generate features and detection decisions on thecurrent frame. Preferably, at the same time, the CPU process starts theFPGA capture and process activity on the FPGA 124. While the CPU isprocessing features for the current frame, the FPGA is computing imagedata for the next frame. The detection and control activity sends thesafety and activate signals out through the FPGA serial I/O interface.The CPU feature and detection processing takes longer than the FPGAcomputations. When the CPU finishes the current frame, the FPGA data isretrieved to the opposite bank (e.g., Bank 1 if processing Bank 0).Diagnostic messages can be output at the end of each frame processing,as well as at any point in the video processing. The process then loopsto set Bank U to the new current bank (Bank 0 or Bank 1), and the FPGAis again initiated.

Turning to FIG. 22, a simplified block diagram is provided of anembodiment of a system resources allocation method, or ping/pongcontrol, in accordance with the present invention. As stated previously,the video processing system 110 includes an FPGA 124 for imageprocessing. This results in the production of derived image data 230comprising edge segmentation, motion segmentation, and regionsegmentation.

Once the derived image data 230 is produced, it is preferably storedwithin one of a plurality of memory banks 230 a,230 b and then provided,via switching, for feature processing. Accordingly, the derived imagedata 230 provided to the feature processing module 214 is static.However, the FPGA 124 continuously processes the electronic pixel data218 and loads the results of the image processing, via switching, intothe memory bank not currently accessible to the processing module 214.Accordingly, the derived image data 230 within the memory banks isaccessible to the feature processing module via switched between thememory banks 230 a,230 b on a first-in-first-out basis.

Preferably, two memory banks 230 a and 230 b are provided. Turning backto FIG. 19, the get next frame step 252 provides for the capture andprocessing of electronic pixel data 218 by the image processing module212 within the FPGA 124. In particular, a control and status register(CSR) is used for selection of the memory banks 230 a,230 b and to setcapture & process bit.

The retrieve FPGA data step 254 provides for obtaining the static datawithin the memory banks for processing of the static data during thecalculating all features step 256. In particular, temporary storageregisters and counters are reset, and the static data is unpacked toprovide the derived image data 230 for processing by the featureprocessing module 214 (FIG. 16).

In an embodiment, and as explained in detail further herein, the featureprocessing module 214 (FIG. 16) performs the calculate all features step256, in response to the derived image data 230 (FIG. 16) and parameterdata 228 (FIG. 16). In particular, the calculate all features step 256produces feature data 232 (FIG. 16) comprising edge or P features,motion features, shadow and lightbeam or region features, and framefeatures.

Further, the detection processing module 216 performs the calculatedetection & control step 258, in response to the feature data 232 (FIG.16) and parameter data 228 (FIG. 16). In particular, frame analysisfault flags are evaluated, detection logic is executed, reference frameupdate requires are evaluated, automatic adaptive thresholds areevaluated, and motion reference data is updated.

The save FPGA data step 260 occurs once the FPGA 124 (FIGS. 15 a and 15b) provides an interrupt to the CPU 125 (FIGS. 15 a and 15 b) indicatingthat the FPGA has completed processing of another image frame and thedata is ready for processing by the feature processing module (FIG. 16). Moreover, if desired, the write diagnosis step 262, which isperformed by the CPU 125 (FIGS. 15 a and 15 b), can store within memoryone or more messages regarding how the processing of the previous framedata progressed. The video processing loop 250 the preferably continuesback to the get next frame step 252.

Turning to FIG. 23, a simplified block diagram is provided of anembodiment of an initialization procedure for the ping-pong system ofFIG. 22. In an embodiment, the ping-pong initialization procedure 264includes an initialize ping-pong addresses module 265, a capture videobuffer module 266, a capture frame module 267, an initiate videoreferences module 268, a process video module 269, a process framemodule 270, and a retrieve FPGA data module 271.

FIG. 24 provides a simplified block diagram of the ping-pong processloop 272 for the ping-pong system of FIG. 22. The top of the loop 272shows the CPU activity while the bottom shows the FPGA activity (not intime scale), with the associated serial I/O and CSR messaging.

FIGS. 25 and 26 illustrate the details of the ping/pong activity(initialization & video loop) in an alternative embodiment having anautomatic contrast circuit (ACC). The ACC can be used to improve systemdetection performance due to slowly changing lighting conditions. TheACC does this by changing the video gain in response to imagecharacteristic criteria and time dynamic criteria. The ACC maximizesgain while preventing too much image white saturation. After a gainchange, the video system is reinitialized.

The ACC, when enabled by user input, functions during initialization tofind the best starting gain by iterating and testing the image result.When a gain is found which satisfies established criteria, iteratingstops, and the process continues to the video loop with the selectedgain. The ACC also functions at the beginning of the video loop, butdoes not iterate to fine a satisfactory gain. Only a single gain changeis performed in the loop per frame. The gain change and consequent videosystem initialization take a much shorter time than a frame time (100ms). The decision to require a gain change in the video loop iscontrolled by criteria calculated in the detection and control portionof the CPU activity. The criteria can include aging, zone activity, andlong and short time-constant filters.

FIG. 27 provides a simplified block diagram of the zone initializationsequence 240 for the system initialization of FIG. 17. The zoneinitialization sequence 240 results in the building of zones in realworld coordinates, generating control zone submasks and constructingcontrol zone masks. FIG. 28 provides a simplified block diagram of thethreshold tables initialization sequence 238 for the systeminitialization of FIG. 17. The threshold tables initialization sequence238 result in the initialization of camera intrinsic parameters, theresolution model, and the object model.

Turning to FIG. 29 (and referencing FIGS. 48 and 49), a simplified blockdiagram is provided of the image processing module for the processingsystem of FIG. 16 which includes an edge detector 301. The edge detector301 preferably includes a modified Sobel operator module 302, a positivedifference module 304, a threshold module 306, an erode module 308, anda label I module 310.

In an embodiment, the modified Sobel operator module 302 receivescurrent (B) image input 312 and generates the edge image (GEB) 314 fromthe current input image. A reference image (GER) 316, initialized in theCPU, is subtracted from the current edge image in the positivedifference operator module 304, where negative values are set to zero.The grey-level edge image is thresholded 306, eroded 308, and labeled310. The output of the label I operator 310 is a 16-bit labeled image318, an equivalence table 320, and counts of the number of labels used322 and the number of entries in the equivalence table. Counts of thenumber of set pixels in the binary input 324 and output 326 of the erodeoperator 308 are also output to the CPU, completing the edge imageprocessing.

Label I operator 310 is used in each thread of the image processing.Label I 310 is the first part of a two step process used to produce thelabeling of the connected components of the binary input. Label 1310passes a 2×2 kernel over the binary input image beginning with the upperleft of the image. The elements of the kernel are identified as follows:

B C A XIf the binary pixel in X is zero, the output is zero. If X is set, thelabels B, A, C are scanned in that order. If all of B, A, C arenon-zero, the next value of a label counter is output at X and thecounter is incremented. If any B, A, C are non-zero, the label operatoris the value of the first non-zero label. If more than one of B, A, C isnon-zero, the first non-zero value is output. If any of the remainingnon-zero labels is different from the output value, the output value andthe different value are written to an equivalence table.

Turning to FIG. 30 (and referencing FIGS. 48 and 49), a simplified blockdiagram of the image processing module for the processing system of FIG.16, having a motion detector that uses regions 340. The system 340preferably includes positive difference modules 342 and 344, thresholdmodules 346 and 348, and dilate modules 350 and 352 for both the current(B) input image (n) and previous (A) image (n−1) respectively. Theoutput is passed to the inclusive OR module 354, erode module 356, andLabel I module 358.

The positive difference of the current grey-level input image (B) 360and the previous image (A) 362 is thresholded 346 and 348 and dilated350 and 352, as well as the positive difference of A and B 342 and 344.The results are inclusively ORed 354. The resulting binary image islabeled as in the edge case (FIG. 29), and the results are passed to theCPU.

The grey-level edge image is thresholded, eroded 356, and labeled 358.The output of the label I operator 358 is a 16-bit labeled image 364, anequivalence table 366, and counts of the number of labels used 368 andthe number of entries in the equivalence table. Counts of the number ofset pixels in the binary input 370 and output 372 of the erode operator356 are also output to the CPU, completing the motion detector imageprocessing using regions.

Label I operator 358 is used in each thread of the image processing.Label I 358 is the first part of a two step process used to produce thelabeling of the connected components of the binary input. Label I 358passes a 2×2 kernel over the binary input image beginning with the upperleft of the image. The elements of the kernel are identified as follows:

B C A XIf the binary pixel in X is zero, the output is zero. If X is set, thelabels B, A, C are scanned in that order. If all of B, A, C arenon-zero, the next value of a label counter is output at X and thecounter is incremented. If any B, A, C are non-zero, the label operatoris the value of the first non-zero label. If more than one of B, A, C isnon-zero, the first non-zero value is output. If any of the remainingnon-zero labels is different from the output value, the output value andthe different value are written to an equivalence table.

Turning to FIG. 31 (and referencing FIGS. 48 and 49), a simplified blockdiagram of the image processing module for the processing system of FIG.16, having region analysis for shadow and lightbeam processing. Thesystem 380 preferably includes positive difference modules 382 and 384,threshold modules 386 and 388, and dilate modules 390 and 392 for boththe current (B) input image 394 and reference (R) input 396,respectively. The output is passed to the inclusive OR module 396, erodemodule 400, and Label I module 402. The output of the label I operator402 is a 16-bit labeled image 404, an equivalence table 406, and countsof the number of labels used 408 and the number of entries in theequivalence table. Counts of the number of set pixels in the binaryinput 410 and output 412 of the erode operator 402 are also output tothe CPU, completing the motion detector image processing using regions.

The system image processing region analysis detection operation isanalogous to the motion detection operation of FIG. 30 except thatinstead of using the immediately previous image input A, it uses aprevious image called the reference image 396 which is updated on CPUcommand as a copy of the current input image. The region analysis threadalso produces a difference image 414 and a 16-level histogram 416 forCPU use.

Turning to FIG. 32 (and referencing FIGS. 48 and 49), a simplified blockdiagram of the image processing module for the processing system of FIG.16, having a motion detector that uses edges 420. The system 420illustrates a selectable alternative motion detection image processingoperation similar to the edge detection operation of FIG. 29, exceptthat it takes the current edge image (GEB) 422 and the previous edgeimage (GEA) 424 as input to the positive difference module 426. Thepositive difference of GEB 422 minus GEA 424 is thresholded 428, eroded430 and labeled 432 as in FIGS. 19, 20, and 21.

The output of the label I operator 432 is a 16-bit labeled image 434, anequivalence table 436, and counts of the number of labels used 438 andthe number of entries in the equivalence table. Counts of the number ofset pixels in the binary input 440 and output 442 of the erode operator430 are also output to the CPU, completing the system image processinghaving a motion detector that uses edges.

Turning to FIG. 33, a simplified block diagram of the feature processingmodule 450 for the processing system of FIG. 16, for calculatingpresence, motion, frame and region features. The system data processing,calculating all features, is performed in the FPGA to unburden the CPUand achieve the desired processing rate.

The presence (P) or edge feature module 452 and the shadow and lightbeam(SL) or region feature module 458 calculations are quite similar to thepoint of generating the edge/region score discounts. Moreover, withinthe P feature 452 and SL feature 458 calculations, the globalcalculations are very similar to the zone calculations. The zonecalculations restrict the spatial range of feature calculations for eachzone using the associated zone mask. The results of the P and SL featurecalculations are stored in a database (feature tables) for use indetection determination 460.

Frame features 456 are calculated differently than P and SL features.Frame features 456 are not features of objects, but of the input greylevel image and the current edge image. Frame statistics are computed inorder to draw inferences about conditions of the camera and video systemintegrity. Frame statistics are also used to condition some detectionvariables that act as adaptive thresholds. Three fault flags can be setby the calculate frame features module 456: illumination fault flag,obscure fault flag, and ajar fault 462. Each of these faults 462 isdetermined through associated metrics. The illumination fault iscontrolled by evaluating the modified Kuiper statistic, the uniformcentered mean, and the variance of the grey level input. The obscure andajar faults use the current and archive edges to detect whether thecamera and/or video system have become obscured or knocked ajar.

The system 450 will not update the reference if any motion is detectedin any zone. To determine if there is motion in each zone, the labelededge image counts the number of non-zero pixels in the labeled zone,calculated in the presence P feature module 452. The non-zero pixelsbecome motion pixels, calculated in the motion M feature module 454. Thesystem 450 counts the non-zero pixels in the motion labeled image toverify if the zone motion pixels in each zone is greater than zero (0).The system 450 counts the non-zero pixels in the zone detection mask foraccumulation in the count.

Turning to FIG. 34, a simplified block diagram of the feature generationsystem 470 of FIG. 33, having a label module 472, a calculate globalpresence features module 474 and a calculate zone presence featuresmodule 476.

The label module 472 receives presence input in the form of labeled edgeimage 478, equivalence tables 480, and label and conflict counts 482.The label module 472 resolves pixel labeling conflicts within theregion, it replaces labels with region numbers, it makes an area call,renumbering regions with sequential indices, and re-indexes the regionagain, passing data related to the number of regions and regions imageto the calculate global presence features model 474.

The calculate global presence features model 474 uses the regions image484, the number of regions 486 and current edges (GEB) 488 to create aglobal feature table. The global feature table is first initialized,regions are labeled as to area, mean grey level intensity, histogram,and centroid. The region is then recalculated for variance of grey leveland centroid, listing the features (global, safety, . . . ) of thepixels within the region.

The calculate P feature zones module 476 takes the aforementionedregions image 484, number of regions 486, the current edges (GEB) 488and creates a zone feature table using zone mask and rectangle 490. Thesystem 470 determines motion in zones by calculating detection in asafety zone, a secondary safety zone, a door zone, a first activationzone, a second activation zone, a first guard zone, and a second guardzone.

Turning to FIG. 35, a simplified block diagram of the calculate presence(edge) feature system of FIG. 34, for calculating global presencefeatures module 500.

The global P features are calculated first by initializing the edgecounts of the feature table 502. The global extent of the image iscalculated in the image first pass module 504. Area, centroid, mean,histogram and edge counts are accumulated and put through the imagesecond pass module 506 where a second central moments and variance isaccumulated. The feature table pass module 508 calculates the derivedfeatures including the spread, elongation, orientation, and ellipseshape of the region. The calculate region scores module 510 determinesdoor rejection, edge shape suppression, and edge grey level suppression.A score comes from the grey level variance of the region and a discountis applied to the score. After region scores are calculated 510, thenext region is looped through the calculations of the feature table passmodule 508.

Turning to FIG. 36, a simplified block diagram of the calculate presencefeature system of FIG. 34, for calculating zone presence features.

The zone presence P features are calculated first by initializing thezone counts of the feature table 522. The global extent of the zone iscalculated in the zone first pass module 524. The zone is calculated todetermine if pixels are in the selected zone mask. Area, centroid, mean,and histogram are also accumulated and put through the image second passmodule 526 where a second central moments and variance is accumulated.The feature table pass module 528 calculates the derived featuresincluding the spread, elongation, orientation, and ellipse shape of theregion. The calculate region scores module 530 determines doorrejection, area proportion suppression, edge shape suppression, and edgegrey level suppression. After region scores are calculated 530, the nextregion is looped through the calculations of the feature table passmodule 528.

Turning to FIG. 37, a simplified block diagram of the feature generationsystem 540 of FIG. 33, having a label module, a calculate global shadowand lightbeam (SL) features module, and a calculate shadow and lightbeamzone features module.

The label module 542 receives presence input in the form of labeled edgeimage 544, equivalence tables 546, and label and conflict counts 548.The label module 542 resolves pixel labeling conflicts within theregion, it replaces labels with region numbers, it makes an area call,renumbering regions with sequential indices, and re-indexes the regionagain, passing data related to the number of regions and regions imageto the calculate global presence features model 550.

The calculate global presence features model 550 uses the regions image552, the number of regions 554 and the current difference image tocreate a global feature table. The global feature table is firstinitialized, regions are labeled as to area, mean grey level intensity,histogram, and centroid. The region image is then recalculated forvariance of grey level and centroid second movements, listing the shapefeatures of the image within the region.

The calculate SL feature zones module 558 takes the aforementionedregions image 552, number of regions 554, the current edges (GEB) 560and creates a zone feature table using zone mask and rectangle 562. Thesystem 540 determines motion in zones by calculating detection in asafety zone, a secondary safety zone, a door zone, a first activationzone, a second activation zone, a first guard zone, and a second guardzone.

Turning to FIG. 38, a simplified block diagram of the calculate shadowand lightbeam region features 570 system of FIG. 37, for calculatingglobal shadow and lightbeam (SL) features.

The global SL features are calculated first by initializing the edgecounts of the feature table 572. The global extent of the image iscalculated in the image first pass module 574. Area, centroid, mean,histogram and edge counts are accumulated and put through the imagesecond pass module 576 where a second central moments and variance isaccumulated. The feature table pass module 578 calculates the derivedfeatures including the spread, elongation, orientation, ellipse shapefactor of the region, modified kniper statistic and mapped mean andvariance. The calculate region scores module 580 determines the SL scorewith region suppression from shadow and light beam discount, shapediscount, and area discount and with transient suppression. After regionscores are calculated 580, the next region is looped through thecalculations of the feature table pass module 578.

Turning to FIG. 39, a simplified block diagram of the calculate shadowand lightbeam region features 590 system of FIG. 37, for calculatingshadow and lightbeam (SL) zone features.

The zone SL features are calculated first by initializing the zonecounts of the feature table 592. The global extent of the zone iscalculated in the zone first pass module 594. The zone is calculated todetermine if pixels or zone rectangle are in the selected zone mask.Area, centroid, mean, and histogram are also accumulated and put throughthe image second pass module 596 where a second central moments andvariance is accumulated. The feature table pass module 598 calculatesthe derived features including the spread, elongation, orientation,ellipse shape factor, modified kniper statistic, and mapped means andvariance of the region. The calculate region scores module 600determines the SL score with region suppression from shadow and lightbeam discount, shape discount, and area discount and with transientsuppression. After region scores are calculated 600, the next region islooped through the calculations of the feature table pass module 598.

FIG. 40 provides a simplified block diagram of a split histogram greylevel analysis 700 for the processing system of FIG. 16, and inparticular the feature processing module 214. In an embodiment, thesplit histogram grey level analysis 700 can be provided to assists inthe determination of region features.

FIG. 41 provides a simplified block diagram of the feature generationsystem 750 of FIG. 33, for calculating frame features. The system 750include a frame analysis initialization module 752, a compute framestatistics module 754, an interpret frame illumination metrics module756, an interpret obscure metrics module 758, an interpret ajar metricsmodule 760, and an update FA filter 762.

FIG. 42 is a simplified block diagram of the information processingmodule of FIG. 16, for detecting the presence of an object from featuresgenerated. The detection sequence includes an evaluate fault flagsmodule 800, an evaluate reference updates module 802, an automaticadaptive thresholds module 804, an update frame analysis data module806, and an update motion reference module 808.

FIG. 43 is a simplified block diagram of the information processingmodule of FIG. 16, for evaluating and updating reference images. Theevaluate reference updates sequence includes a zones motion detectionmodule 822, an update update counters module 824, a decide standardupdate module 826, a decide archive update module 828, a decide gainchange module 830, and a decide learn-outs module 832.

FIG. 44 is a simplified block diagram of the information processingmodule of FIG. 16, for changing threshold values relative to changingbackground values from the field of view. The automatic adaptivethresholds sequence includes an initialize SL thresholds edge threshold840, a calculate zone statistics module 842, a calculate zone metricsmodule 844, and an apply metrics module 846.

FIG. 45 is a simplified block diagram of the information processingmodule of FIG. 16, for determining the geometric association of edge andzone data in a detected object. The sequence includes an initializemodule 850, an application for each qualified edge region 852, anapplication regarding initialization for a particular region 854, and atraverse region bounding rectangle module 856. The sequence thencontinues to FIG. 46.

FIG. 46 is a continuation of FIG. 45 of the simplified block diagram ofthe information processing module of FIG. 16, and includes testingregion edges to determine zone intersections in detected objects. Thesequence includes a test edge region/zone intersection modules 862,864and a test region/motion/zone intersection module 866.

FIG. 47 is a continuation of FIG. 46 of the simplified block diagram ofthe information processing module of FIG. 16, and includes evaluatingregion scores of zones to determine zone intersections in detectedobjects. The sequence includes evaluate region scores modules 872,874,set derived flags update counter/histories module 876, and an update topdetection scores & safety zone latch module 878.

In an embodiment, an automatic door control and safety system isprovided that controls door behavior in accordance with logic thatinterprets a nominally optically sensed object situation and environmentproximate to the door. The system uses a camera sensor sub-system fittedwith an appropriate lens in order to generate an image of the desiredsensing area. Digital images produced by the camera sub-system areprocessed using image processing in a processing sub-system in order todevelop data used to drive specific decision logic to effect desireddoor control. Thus, door control is effected by computer interpretationof image content.

In an embodiment, from a processing point of view, the systemincorporates several processing stages: 1) image formation; 2) imageconditioning; 3) image processing; 4) image content processing; 5)derived data processing; 6) data interpretation processing; and 7)control logic processing.

The door control and safety system is supported by hardware elements toinclude the camera sub-system, and a general purpose processorsub-system that can be augmented by a digital signal processing device.The camera sub-system can include a lens system, a charge-coupled deviceimaging device, amplifiers, and an analog-to-digital conversion element.These element can be commonly found together in home computerapplications, for example, which interface a digital camera to producedigital images on the computer screen for capture and storage for avariety of purposes.

The system uses a selection of image processing operators, implementedin an algorithm, and subsequent derived data processing andinterpretation. The selected image processing operators and imagecontent processing are derived through the optical phenomena exhibitedby objects within the field of view of the camera. The image processingoperates on the numbers contained in the array representative of scenedetermined though the lens and camera mounting geometry. This imageprocessing creates internal arrays of numbers which are the results ofthe image processing, to be used by subsequent operations thus forming asequence of image processing operations.

In an embodiment of the system, the entire image field is processed.Furthermore, there are no prior assumptions about target objects used todevelop any processing elements designed to match anticipated objectcharacteristics for the purpose of selecting subsets of the entire imagefield.

At the beginning of the image processing sequence, the image processingaccepts a new input image of the scene (which is a single time sample(“frame”) of the on-going image digitization stream). Storage isprovided in order to maintain a previous image frame for comparison to anewly captured image frame (a “background” image). This stored imageframe is captured in the same way as a new frame, and, in particular, isa single image frame, not an average of more than one frame.

In an embodiment, each new image frame is filtered to remove specklenoise using a median filter. The median filter removes isolated noisewhile not blurring the image as does averaging. Such isolation noise maybe due to imaging sensor noise, downstream electronics noise orenvironmentally-produced scintillation. The image stored for comparisonis filtered one with the median filter, as is the current image. Themedian filter in can be implemented as a 3×3 filter kernel that ispassed over every pixel in the image array. The value at the center ofthe kernel is deposited in a new image array, and the value is thatwhich is the median of the nine numbers in the filter kernel.

After image filtering, two new image arrays are generated (i.e., FIGS.52 and 53). The first new image array (FIG. 52) is determined as thepixel-by-pixel difference of the current image minus the backgroundimage (“positive contrast”). The second new image array (FIG. 53) isdetermined as the pixel-by-pixel difference fo the background imageminus the current image (“negative contrast”). The images are maintainedas arrays of 8-bit numbers, so that when difference values are greaterthan 255 or less than 0, values are clipped accordingly.

After differencing, the images still contain 8-bit values. (Images withmultiple bit levels are commonly referred to as grey-scale images).After image differencing, a thresholding operator is applied to each ofthe resulting positive and negative contrast grey-scale images. Thethreshold values applied to the two images may be different. The valuescan be fixed or adaptive wherein changes are made based on downstreamimage interpretation results. The pixel-by-pixel thresholding operationproduces two new images. For each image, when the grey level in theinput image exceeds the associated threshold value, a “1” is placed inthe output image array, otherwise a “0” is placed. The result of thethresholding operation is thus two “binary” images.

Turning to FIGS. 54 and 55, selected binary image processing techniquesof mathematical morphology are applied to the binary images tofacilitate downstream image interpretation. In an embodiment, operatorsare selected to remove isolated binary regions that could not be fromsignificant objects, while improving the “connectedness” of largerregions that may be significant. Referred to as shape filtering, each ofthe two binary images are filtered similarly to the median filtermechanism (a 3×3 spatial kernel), except that the filter kerneloperation is a maximum operator followed by a minimum operation, not themedian operation. Such a filter is referred to as a binary closing or“close.” A “close” is a “dilation” followed by an “erosion.” The“dilation” is the maximum operation on the kernel, and the “erosion” isthe minimum operation.

Turning to FIG. 56, the two closed binary images (FIGS. 54 and 55) arelogically OR-ed pixel-by-pixel to produce a resultant binary imagerepresentative of both positive and negative contrast differences withrespect to the input images.

Turning to FIG. 57, a connected components algorithm is applied to theresultant binary OR image (FIG. 56). This algorithm identifies all theconnected binary regions in the image. A connected region is one whereinevery member pixel is a neighbor of at least one other member pixel. Theconnected components algorithm labels each region and builds a databasecontaining derived features of each region. In an embodiment, thefeatures can include region area, bounding rectangle, circularity,ellipse major and minor axis lengths, and perimeter. The region featuredata is processed to select regions of interest. The regions are adirect result of the presence of the object in the field of view. Nooperator selection of sub-image regions of the total image field isinvolved in selecting the object-related regions—the regions aredetermined by the object.

With a database representative of image content, the features of eachregion are considered by interpretation logic to develop control logicdecisions. In an embodiment, the interpretation logic is implemented asa set of “if-then-else” constructs, and can utilize arithmeticcombination of the basic region features in order to determine imagecontent interpretation. For instance, the resulting region area can beused to infer the presence of an object of interest, and the regioncentroid and bounding rectangle determine the location of that object.(The bounding rectangle is the smallest rectangle that includes allpixels belonging to the region.)

In an embodiment, the operator can define rectangular regions of theimage field of view to determine areas for specific control actions. Thebounding rectangle coordinates of the computer-derived object regions ofinterest are compared to the coordinates of the operator-determineddecision regions in order to determine subsequent control logic results.If an object is declared to be in the safety zone, for example, thecontrol logic indicates that the door should remain open until thesafety zone is clear. Similarly, if an object is determined to be in theactivation zone (the binary region bounding rectangle representative ofthe image object intersects the activation zone decision rectangle),then the signal is sent to open the door. In an embodiment, the imageregions selected by the operator for control logic purposes are not usedin any way to initialize or otherwise influence the image processing ofthe entire image in order to determine image content.

While the specific embodiments have been illustrated and described,numerous modifications come to mind without significantly departing fromthe spirit of the invention, and the scope of protection is only limitedby the scope of the accompanying claims.

1. An input device for communication with a controller for an automaticdoor comprising: a first sequencing key which is configured to prompt auser to enter a first set of data into the device when actuated a firsttime and prompt the user to enter a second set of data when actuated asecond time; at least one input key; at least one input; and, a displayfor displaying user input and pre-stored user options for selection inresponse to prompts from the first sequencing key.
 2. The device ofclaim 1 wherein the display displaying only a portion of the pre-storeddata to be displayed for each actuation of the at least one input key.3. The device of claim 1 including an input selector which accepts andstores user input when actuated.
 4. The device of claim 1 including asecond sequencing key which is configured to prompt a user to enter athird set of data into the device using a first display key whenactuated a first time and prompt the user to enter a fourth set of datainto the device using the first display key when actuated a second time.5. The device of claim 4 including a third sequencing key which isconfigured to prompt a user to enter a fifth set of data into the deviceusing a first display key when actuated a first time and prompt the userto enter a sixth set of data into the device using the first display keywhen actuated a second time.
 6. The device of claim 5 wherein the firstand third sequencing keys are configured with a hierarchy, the hierarchyallowing the first sequencing key to override operation of the thirdsequencing key.
 7. The device of claim 5 wherein the first, second andthird sequencing keys are configured with a hierarchy, the hierarchyallowing the first sequencing key to override the operation of thesecond and third sequencing keys.
 8. The device of claim 4 wherein thefirst and second sequencing keys are configured with a hierarchy, thehierarchy allowing the first sequencing key to override operation of thesecond sequencing key.
 9. The device of claim 1 including a secondsequencing key which is configured to prompt a user to enter a third setof data into the device using the first display key when actuated afirst time and prompt the user to enter a fourth set of data into thedevice using a second display key when actuated a second time.
 10. Thedevice of claim 1 including a third sequencing key which is configuredto prompt a user to enter a fifth set of data into the device using afirst display key when actuated a first time and prompt the user toenter a sixth set of data into the device using a second display keywhen actuated a second time.
 11. The device of claim 1 wherein the atleast one input is a numerical input.
 12. The device of claim 1 whereinthe at least one input is an alpha input.
 13. A sensor system forcontrolling an automatic door which has a door panel selectivelyblocking an opening comprising: a sensor having a field of view of areasof interest about the opening and a signal output relative to objectssensed in the field of view of the sensor; a signal processor responsiveto sensor output signals; a door drive responsive to the signalprocessor; and, an input device having a signal output, the signalprocessor responsive to output signals from the input device, whereinthe input device having a pose input for permitting input dataindicative of the pose of the sensor as mounted to obtain the field ofview, the pose input including at least one of the group consisting ofinstalled height of the sensor, width of the opening, and distance ofthe sensor from the opening.
 14. The sensor system of claim 13 whereinthe pose input includes a nominal height of the sensor.
 15. The sensorsystem of claim 13 wherein the pose input includes a model number of thedoor associated with the field of view.
 16. The sensor system of claim13 wherein the pose input includes the side of the opening on which thesensor is installed.
 17. The sensor system of claim 13 wherein the poseinput includes a length of the opening.
 18. The sensor system of claim13 wherein the pose input includes a first position of an activationzone.
 19. The sensor system of claim 13 wherein the pose input includesa second position of an activation zone.
 20. The system of claim 13wherein the sensor is a camera.
 21. The sensor system of claim 13wherein the sensor signal output includes object boundary coordinates.22. The sensor system of claim 13 wherein the signal processor isselected from at least one of the group consisting of CPUs, FPGAs, andvideo digitizers.
 23. A sensor system for controlling an automatic doorwhich has a door panel selectively blocking an opening, the sensorsystem comprising: a sensor configured to sense objects in a field ofview; and, an input device having an output for communication with acontroller for the automatic door, and an input relative to at least oneof the group consisting of installed sensor height, width of theopening, and distance of the sensor from the opening.
 24. The system ofclaim 23 wherein the input device having a data set stored relative todoor model.
 25. The system of claim 23 wherein the sensor is a camera.26. A system for controlling an automatic door which selectively blocksan opening, the system comprising: a sensor configured and adapted tosense one of either active or passive energy or both from a field ofview; an imager configured to form an image from the energy sensed bythe sensor; an image analyzer responsive to the imager, the imageanalyzer being configured and adapted to define image contentinformation from the image; a decision maker responsive to the imageanalyzer, the decision maker being adapted and configured to makedeterminations about the objects in the field of view based upon theimage content information, and having a first decision metric, the firstdecision metric being based upon the relationship of selected imageinformation content to a first predetermined area of zone within theimage; and, a door controller for controlling at least the opening andclosing of the door, the door controller being responsive to thedecision maker.
 27. The system of claim 26 wherein the decision makerhaving a first decision metric, the first decision metric being basedupon the relationship of selected image information content to a secondpredetermined area or zone within the image.
 28. The system of claim 27wherein the selected image information content is indicative of thepresence of a stationary object.
 29. The system of claim 28 wherein therelationship to the area or zone includes whether at least a portion ofthe object is within the first or second predetermined zones.
 30. Thesystem of claim 27 wherein the relationship to the area or zone includeswhether at least a portion of an object is within the first or secondpredetermined zones.
 31. The system of claim 26 including a geometrymapper which translates the dimensions and geometries of the field ofview to coordinates.
 32. The system of claim 31 wherein the coordinatesrepresent the edges of the objects in the field of view.
 33. The systemof claim 31 wherein the image content information includes polarcoordinates related to an object in the field of view.
 34. The system ofclaim 26 wherein the sensor is a camera.
 35. The system of claim 26wherein the energy sensed by the sensor to form an image includesboundary information.
 36. The system of claim 35 wherein the boundaryinformation includes boundary coordinates representing edges of anobject.
 37. The system of claim 26 wherein the image content informationincludes boundary coordinates.
 38. The system of claim 37 wherein theboundary coordinates represent edges of an object.
 39. The system ofclaim 26 wherein the imager is selected from at least one of the groupconsisting of CPUs, FPGAs, signal processing devices and videodigitizers.
 40. The system of claim 26 wherein the image analyzer isselected from at least one of the group consisting of CPUs, FPGAs,signal processing devices and video digitizers.
 41. The system of claim26 wherein the decision maker selected from at least one of the groupconsisting of CPUs, FPGAs, signal processing devices and videodigitizers.
 42. A system for controlling a door comprising: a sensor forcollecting image data; a control unit receiving the data from thesensor, wherein the control unit has a means for defining at least aportion of an image as a control zone; a drive motor for controlling theopening and closing of the door, the drive motor receiving controlsignals from the control unit; and, a first control zone whereinselected image data within the first control zone will cause a controlsignal to open the door.
 43. The system of claim 42 including a secondcontrol zone wherein selected image data within the second control zonewill generate a first control condition preventing the door fromclosing.
 44. The system of claim 43 wherein the control unit analyzesthe image data in the second control zone at a first time; the controlunit analyzes the image data in the second control zone at a secondtime; and the control unit compares the image data at the first time andthe image data at the second time to determine whether an object ispresent in the first control zone.
 45. The system of claim 43 furtherincluding a third control zone wherein selected image data within thethird control zone will generate a second control condition preventingthe door from closing.
 46. The system of claim 43 wherein the secondcontrol zone comprises a plurality of predetermined shapes.
 47. Thesystem of claim 46 wherein the third control zone comprises a pluralityof predetermined shapes, the shapes of the third control zone beingcomplementary to the shapes of the second control zone.
 48. The systemof claim 47 wherein the control unit analyzes the image data in thethird control zone at a first time; the control unit analyzes the imagedata in the third control zone at a second time; and the control unitcompares the image data at the first time and the image data at thesecond time to determine whether an object is present in the thirdcontrol zone.
 49. The door of claim 42 wherein the means for definingincludes defining the image as a pixel map and choosing coordinates fromall pixel coordinates by direct access within the control zone.
 50. Thesystem of claim 42 wherein the means for defining includes choosing frommultiple predefined zones.
 51. The system of claim 42 wherein the meansfor defining includes putting real objects in a field of view so as todelineate boundary coordinates and the real objects become part of theimage data, the control zone being defined from the real objects imagedata.
 52. The system of claim 42 wherein the control unit analyzes theimage data in the first control zone at a first time; the control unitanalyzes the image data in the first control zone at a second time; andthe control unit compares the image data at the first time and the imagedata at the second time to determine whether an object is present in thefirst control zone.
 53. The system of claim 52 wherein the object isanalyzed to determine if any portion of the object is within the firstcontrol zone.
 54. The system of claim 53 wherein the selected image datawithin the first control zone causes the control signal to open the doorif any portion of the object is within the first control zone.
 55. Thesystem of claim 54 wherein the control unit chooses a portion of theobject to determine a bottom edge of the object.
 56. The system of claim55 wherein the control unit analyzes the bottom edge to determine if thebottom edge is within the first control zone.
 57. The system of claim 56wherein a fourth control zone is defined adjacent the first controlzone, the fourth control zone possessing objects between the firstcontrol zone and the sensor.
 58. The system of claim 57 wherein thefourth control zone is between the first control zone and the camera.59. The system of claim 58 wherein the presence of the object in thefourth control zone will prevent the first control zone from sending thecontrol signal to open the door.
 60. The system of claim 59 wherein thepresence of the object in the fourth control zone will cause the firstcontrol zone to send the control signal to open the door.
 61. The systemof claim 42 wherein the first control zone includes means for comparingthe image data to a plurality of user specified dimensions.
 62. Thesystem of claim 42 wherein the sensor senses energy selected from thegroup consisting of: visible light waves, infrared lights waves,microwaves, radar, laser, and sound waves.
 63. The system of claim 42wherein a user can enter data to define at least one control zoneparameter from the group consisting of area, location, shape, number ofcontrol zones and control criteria.
 64. The system of claim 42 whereinthe sensor is a camera.
 65. The system of claim 42 wherein the imagedata includes boundary coordinates.
 66. The system of claim 65 whereinthe boundary coordinates are polar coordinates.
 67. A system forcontrolling a door comprising: a sensor, the sensor having, for example,a beam for sensing coherent energy from objects near the door; a controlunit receiving electronic values from the sensor indicative of energysensed by the camera; a drive motor for controlling the opening andclosing of the door, the drive motor receiving control signals from thecontrol unit; and means for defining a portion of a beam pattern as acontrol zone wherein the control zone is defined within a beam pattern.68. The system of claim 67 wherein the electronic values representingenergy sensed by the sensor are coordinate values.
 69. The system ofclaim 68 wherein the coordinate values represent boundaries of theenergy sensed.